Inline reliability coding for storage on a network

ABSTRACT

This disclosure describes a programmable device, referred to generally as a data processing unit, having multiple processing units for processing streams of information, such as network packets or storage packets. This disclosure also describes techniques that include enabling data durability coding on a network. In some examples, such techniques may involve storing data in fragments across multiple fault domains in a manner that enables efficient recovery of the data using only a subset of the data. Further, this disclosure describes techniques that include applying a unified approach to implementing a variety of durability coding schemes. In some examples, such techniques may involve implementing each of a plurality of durability coding and/or erasure coding schemes using a common matrix approach, and storing, for each durability and/or erasure coding scheme, an appropriate set of matrix coefficients.

TECHNICAL FIELD

This disclosure relates to processing packets of information, forexample, in the fields of networking and storage

BACKGROUND

In a typical cloud-based data center, a large collection ofinterconnected servers provides computing and/or storage capacity forexecution of various applications. For example, a data center maycomprise a facility that hosts applications and services forsubscribers, i.e., customers of the data center. The data center may,for example, host all of the infrastructure equipment, such as computenodes, networking and storage systems, power systems, and environmentalcontrol systems. In most data centers, clusters of storage systems andapplication servers are interconnected via a high-speed switch fabricprovided by one or more tiers of physical network switches and routers.Data centers vary greatly in size, with some public data centerscontaining hundreds of thousands of servers, and are usually distributedacross multiple geographies for redundancy.

Many devices within a computer network, e.g., storage/compute servers,firewalls, intrusion detection devices, switches, routers or othernetwork attached devices, often use general purpose processors,including multi-core processing systems, to process data, such asnetwork or storage data. However, general purpose processing cores andmulti-processing systems are normally not designed for high-capacitynetwork and storage workloads of modern network and can be relativelypoor at performing packet stream processing. Further, in a large scalefabric, storage systems may become unavailable from time to time, due tohardware error, software error, or another reason. Data durabilityprocedures may be employed to ensure access to critical data.

SUMMARY

In general, this disclosure describes a programmable device, referred togenerally as a data processing unit, having multiple processing unitsfor processing streams of information, such as network packets orstorage packets. In some examples, the processing units may beprocessing cores, and in other examples, the processing units may bevirtual processors, hardware threads, hardware blocks, or othersub-processing core units. As described herein, the data processing unitincludes one or more specialized hardware accelerators configured toperform acceleration for various data-processing functions.

This disclosure also describes techniques that include enabling datadurability (or data reliability) coding on a network. In some examples,such techniques may involve the data processing unit storing data infragments across multiple fault domains in a manner that enablesefficient recovery of the data even if only a subset of the data isavailable. Data fragments may be generated, by data durability circuitryincluded within the data processing unit, using one of a variety ofdurability or erasure coding schemes that enable recovery of data whereone or more fragments are unavailable due to software or hardware erroror for another reason, such as maintenance. Data fragments may also bestored, by the data processing unit, across multiple fault domains on anetwork to help reduce the chance that multiple fragments of data areunavailable. Techniques in accordance with one or more aspects of thepresent disclosure may enable inline data recovery of data in the datapath of the storage with low or very low latency, while requiringsignificantly less storage than a simple data replication scheme. Insome examples, one or more hosts or server devices may effectivelyoffload, to one or more data processing units, some or all of thecomputing operations that might otherwise be performed by those hosts orserver devices for purposes of data durability and/or reliability. Byoffloading the performance of such operations to data processing units,one or more of the hosts or server devices may be available to performother tasks and/or operate more efficiently. In some examples, dataprocessing units may, from the perspective of the hosts and/or serverdevices, perform data durability operations (data encoding, datadecoding, and recovery) on network traffic transparently.

This disclosure further describes techniques that include applying aunified approach to implementing a variety of durability coding schemes.In some examples, such techniques may involve implementing each of aplurality of durability coding and/or erasure coding schemes using acommon matrix approach, and storing, for each durability and/or erasurecoding scheme, an appropriate set of matrix coefficients. Suchtechniques may simplify the logic required to implement multipledurability and/or erasure coding schemes. Further, such techniques mayprovide an effective software abstraction layer, enabling commonconfigurations to be used to implement a variety of schemes, therebyproviding reliability, configurability, and flexibility.

In one example, this disclosure describes a method comprisinggenerating, by a device and from a set of data, a plurality of datafragments; storing, by the device, the plurality of data fragments on anetwork, wherein at least some of the plurality of data fragments isstored within a different fault domain on the network; receiving, by thedevice and from a computing device, a request to access at least aportion of the set of data; determining, by the device, that one or moreof the plurality of data fragments is not available on the network;identifying, by the device, a plurality of available data fragments,wherein the plurality of available data fragments is a subset of theplurality of data fragments; retrieving, by the device, each of theplurality of available data fragments over the network; generating, bythe device, a reconstructed set of data from the available datafragments; and responding to the request by outputting, by the device,the reconstructed data.

In another example, this disclosure describes a method comprisingidentifying a reliability scheme from among a plurality of availablereliability schemes, determining, based on the reliability scheme, acoefficient matrix for implementing the reliability scheme; storing thecoefficient matrix; receiving data to be stored; producing, from thedata and by applying the coefficient matrix to the data, parity data;receiving a request for the data; and generating the data based on theparity data, wherein generating the data includes generating the data byapplying the coefficient matrix.

The details of one or more examples of the disclosure are set forth inthe accompanying drawings and the description below. Other features,objects, and advantages of the disclosure will be apparent from thedescription and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example system 8 including oneor more network devices configured to efficiently process a series ofwork units in a multiple core processor system, in accordance with thetechniques of this disclosure.

FIG. 2 is a block diagram illustrating an example data processing unit(DPU) including two or more processing cores, in accordance with thetechniques of this disclosure.

FIG. 3 is a block diagram illustrating another example of a DPUincluding two or more processing clusters, in accordance with thetechniques of this disclosure.

FIG. 4 is a block diagram illustrating an example processing clusterincluding two or more processing cores, in accordance with thetechniques of this disclosure.

FIG. 5 is a block diagram illustrating an example accelerator, inaccordance with one or more aspects of the present disclosure.

FIG. 6A is a conceptual diagram illustrating an example write operationperformed pursuant to a basic replication data reliability scheme, inaccordance with one or more aspects of the present disclosure.

FIG. 6B is a conceptual diagram illustrating an example write operationperformed pursuant to a simple parity encoding data reliability scheme,in accordance with one or more aspects of the present disclosure.

FIG. 6C is a conceptual diagram illustrating an example write operationperformed pursuant to an erasure coding data reliability scheme, inaccordance with one or more aspects of the present disclosure.

FIG. 7A is a conceptual diagram illustrating a matrix operation that maybe used for encoding data pursuant to a Reed-Solomon data reliabilityscheme, in accordance with one or more aspects of the presentdisclosure.

FIG. 7B is a conceptual diagram illustrating a matrix operation that maybe used for decoding data that has been encoded pursuant to aReed-Solomon data reliability scheme, in accordance with one or moreaspects of the present disclosure.

FIG. 7C is a conceptual diagram illustrating a strided read operationthat may be used to generate parity blocks, in accordance with one ormore aspects of the present disclosure.

FIG. 7D is a conceptual diagram illustrating a matrix operation that maybe used for encoding and/or decoding data pursuant to a parity arraycoding scheme, in accordance with one or more aspects of the presentdisclosure.

FIG. 8A is a block diagram illustrating example operations on a networkin which stored data fragments are read from locations on the network,in accordance with one or more aspects of the present disclosure.

FIG. 8B is a block diagram illustrating an example read operation on anetwork in which some data fragments stored on the network areunavailable, in accordance with one or more aspects of the presentdisclosure.

FIG. 9A is a flow diagram illustrating inline reliability codingoperations performed by an example access node in accordance with one ormore aspects of the present disclosure.

FIG. 9B is a flow diagram illustrating flexible data durability codingoperations performed by an example access node in accordance with one ormore aspects of the present disclosure.

DETAILED DESCRIPTION

FIG. 1 is a block diagram illustrating an example system 8 including oneor more network devices configured to efficiently process a series ofwork units in a multiple core processor system. As described herein,techniques for performing data durability coding may provide technicalbenefits that include enabling inline data recovery in a flexible andstorage-efficient manner. Access nodes may also be referred to as dataprocessing units (DPUs), or devices including DPUs, in this disclosure.In the example of FIG. 1, various data structures and processingtechniques are described with respect to access nodes 17 within a datacenter 10. Other devices within a network, such as routers, switches,servers, firewalls, gateways and the like, having multiple coreprocessor systems may readily be configured to utilize the dataprocessing techniques described herein.

Data center 10 represents an example of a system in which varioustechniques described herein may be implemented. In general, data center10 provides an operating environment for applications and services forcustomers 11 coupled to the data center by service provider network 7and gateway device 20. Data center 10 may, for example, hostinfrastructure equipment, such as compute nodes, networking and storagesystems, redundant power supplies, and environmental controls. Serviceprovider network 7 may be coupled to one or more networks administeredby other providers, and may thus form part of a large-scale publicnetwork infrastructure, e.g., the Internet.

In some examples, data center 10 may represent one of manygeographically distributed network data centers. In the example of FIG.1, data center 10 is a facility that provides information services forcustomers 11. Customers 11 may be collective entities such asenterprises and governments or individuals. For example, a network datacenter may host web services for several enterprises and end users.Other exemplary services may include data storage, virtual privatenetworks, file storage services, data mining services, scientific- orsuper-computing services, and so on.

In the illustrated example, data center 10 includes a set of storagesystems and application servers 12 interconnected via a high-speedswitch fabric 14. In some examples, servers 12 are arranged intomultiple different server groups, each including any number of serversup to, for example, n servers 12 ₁-12N. Servers 12 provide computationand storage facilities for applications and data associated withcustomers 11 and may be physical (bare-metal) servers, virtual machinesrunning on physical servers, virtualized containers running on physicalservers, or combinations thereof.

In the example of FIG. 1, each of servers 12 is coupled to switch fabric14 by an access node 17 for processing streams of information, such asnetwork packets or storage packets. In example implementations, accessnodes 17 may be configurable to operate in a standalone networkappliance having one or more access nodes. For example, access nodes 17may be arranged into multiple different access node groups 19, eachincluding any number of access nodes up to, for example, x access nodes17 ₁-17X. In other examples, each access node may be implemented as acomponent (e.g., electronic chip) within a device, such as a computenode, application server, storage server, and may be deployed on amotherboard of the device or within a removable card, such as a storageand/or network interface card.

In general, each access node group 19 may be configured to operate as ahigh-performance I/O hub designed to aggregate and process networkand/or storage I/O for multiple servers 12. As described above, the setof access nodes 17 within each of the access node groups 19 providehighly-programmable, specialized I/O processing circuits for handlingnetworking and communications operations on behalf of servers 12. Inaddition, in some examples, each of access node groups 19 may includestorage devices 27, such as solid state drives (SSDs) and/or hard diskdrives (HDDs), configured to provide network accessible storage for useby applications executing on the servers 12. In some examples, one ormore of the SSDs may comprise non-volatile memory (NVM) or flash memory.

Each access node group 19, including its set of access nodes 17 andstorage devices 27, and the set of servers 12 supported by the accessnodes 17 of that access node group 19 may be referred to herein as anetwork storage compute unit (NCSU) 40. Illustrated in FIG. 1 are NCSU40-1 through NCSU 40-N, which represent any number of NCSUs. In someexamples, data center 10 may include many NCSUs, and multiple NCSUs 40may be organized into logical racks or physical racks within data center10. For example, in some implementations, two NCSUs may compose alogical rack, and four NCSUs may compose a physical rack. Otherarrangements are possible.

In a large scale fabric, storage systems represented by one or moreaccess node groups 19 or NCSUs 40 may become unavailable from time totime. Failure rates of storage systems are often significant, even ifsingle component failure rates are quite small. Further, storage systemsmay become unavailable for reasons other than a software error orhardware malfunction, such as when a storage system or other device isbeing maintained or the software on such a device is being modified orupgraded. Accordingly, as further described herein, data durabilityprocedures may be employed to ensure access to critical data stored on anetwork when one or more storage systems are unavailable.

In some examples, one or more hardware or software subsystems may serveas a failure domain or fault domain for storing data across data center10. For instance, in some examples, a failure domain may be chosen toinclude hardware or software subsystems within data center 10 that arerelatively independent, such that a failure (or unavailability) of onesuch subsystem is relatively unlikely to be correlated with a failure ofanother such subsystem. Storing data fragments in different failuredomains may therefore reduce the likelihood that more than one datafragment will be lost or unavailable at the same time. In some examples,a failure domain may be chosen at the NCSU level, such that each NCSUrepresents a different failure domain. In other examples, failuredomains may be chosen more broadly, so that a failure domain encompassesmore than one NCSU so that a failure domain may encompass a logical orphysical rack comprising multiple NCSUs. Broader or narrower definitionsof a failure domain may also be appropriate in various examples,depending on the nature of the network 8, data center 10, or subsystemswithin data center 10.

As further described herein, in one example, each access node 17 is ahighly programmable I/O processor specially designed for offloadingcertain functions from servers 12. In one example, each access node 17includes a number of internal processor clusters, each including two ormore processing cores and equipped with hardware engines that offloadcryptographic functions, compression and regular expression (RegEx)processing, data durability functions, data storage functions andnetworking operations. In this way, each access node 17 includescomponents for fully implementing and processing network and storagestacks on behalf of one or more servers 12. In addition, access nodes 17may be programmatically configured to serve as a security gateway forits respective servers 12, freeing up the processors of the servers todedicate resources to application workloads. In some exampleimplementations, each access node 17 may be viewed as a networkinterface subsystem that implements full offload of the handling of datapackets (with, in some examples, zero copy in server memory) and storageacceleration for the attached server systems. In one example, eachaccess node 17 may be implemented as one or more application-specificintegrated circuit (ASIC) or other hardware and software components,each supporting a subset of the servers. In accordance with thetechniques of this disclosure, any or all of access nodes 17 may includea data durability or similar accelerator unit. That is, one or morecomputing devices may include an access node including one or more datadurability, data reliability, and/or erasure coding accelerator units,according to the techniques of this disclosure.

The data durability accelerator unit of the access node, according tothe techniques of this disclosure, may be configured to store data infragments across multiple fault domains in a manner that enablesefficient recovery of the data using or based on a subset of the datafragments. When storing data, the data durability accelerator unit mayencode data using any of a variety of data durability or erasure codingschemes that enable recovery of data when one or more of such fragmentsare unavailable due to software or hardware error, or for anotherreason, such as modifications (e.g., software upgrades) being performedon the storage unit where a data fragment is being stored. Further, thedata durability accelerator unit may provide a flexible and/orconfigurable data durability system by applying a unified approach toimplementing a variety of data durability coding schemes. In someexamples, the data durability accelerator may implement multiple datadurability coding schemes or algorithms through a common matrixapproach. In such an example, each data durability scheme or algorithmmay be selected or configured through a different coefficient matrix. Acommon algorithm may be applied that implements, based on values in theselected or configured coefficient matrix, a different data durabilityalgorithm.

In the example of FIG. 1, each access node 17 provides connectivity toswitch fabric 14 for a different group of servers 12 and may be assignedrespective IP addresses and provide routing operations for the servers12 coupled thereto. Access nodes 17 may interface with and utilizeswitch fabric 14 so as to provide full mesh (any-to-any)interconnectivity such that any of servers 12 may communicate packetdata for a given packet flow to any other of the servers using any of anumber of parallel data paths within the data center 10. In addition,access nodes 17 described herein may provide additional services, suchas storage (e.g., integration of solid-state storage devices), security(e.g., encryption), acceleration (e.g., compression), data reliability(e.g., erasure coding), I/O offloading, and the like. In some examples,one or more of access nodes 17 may include storage devices, such ashigh-speed solid-state drives or rotating hard drives, configured toprovide network accessible storage for use by applications executing onthe servers. More details on the data center network architecture andinterconnected access nodes illustrated in FIG. 1 are available in U.S.Provisional Patent Application No. 62/514,583, filed Jun. 2, 2017,entitled “Non-Blocking Any-to-Any Data Center Network with PacketSpraying Over Multiple Alternate Data Paths,”, the entire content ofwhich is incorporated herein by reference.

Two example architectures of access nodes 17 are described below withrespect to FIG. 2, FIG. 3, and FIG. 4. With respect to some or all ofsuch examples, the architecture of each access node 17 comprises amultiple core processor system that represents a high performance,hyper-converged network, storage, and data processor and input/outputhub. The architecture of each access node 17 is optimized for highperformance and high efficiency stream processing.

A stream is defined as an ordered, unidirectional sequence ofcomputational objects that can be of unbounded or undetermined length.In a simple example, a stream originates in a producer and terminates ata consumer, is operated on sequentially, and is flow-controlled. In someexamples, a stream can be defined as a sequence of stream fragments,each representing a portion of data communicated by a stream. In oneexample, a stream fragment may include a memory block contiguouslyaddressable in physical address space, an offset into that block, and avalid length. Streams can be discrete, such as a sequence of packetsreceived from a network, or continuous, such as a stream of bytes readfrom a storage device. A stream of one type may be transformed intoanother type as a result of processing. Independent of the stream type,stream manipulation requires efficient fragment manipulation. Anapplication executing on one of access nodes 17 may operate on a streamin three broad ways: the first is protocol processing, which consists ofoperating on control information or headers within the stream; thesecond is payload processing, which involves significant accessing ofthe data within the stream; and third is some combination of bothcontrol and data access.

Stream processing is a specialized type of conventional general-purposeprocessing supporting specialized limitations with regard to both accessand directionality. Processing typically only accesses a limited portionof the stream at any time, called a “window,” within which it may accessrandom addresses. Objects outside of the window are not accessiblethrough a streaming interface. In contrast, general purpose processingviews the whole memory as randomly accessible at any time. In addition,stream processing generally progresses in one direction, called theforward direction. These characteristics make stream processing amenableto pipelining, as different processors within one of access nodes 17 cansafely access different windows within the stream.

As described herein, data processing units of access nodes 17 mayprocess stream information by managing “work units.” In general, a workunit (WU) is a container that is associated with a stream state and usedto describe (i.e. point to) data within a stream (stored in memory)along with any associated meta-data and operations to be performed onthe data. In the example of FIG. 1, streams of data units maydynamically originate within a peripheral unit of one of access nodes 17(e.g. injected by a networking unit, a host unit, or a solid state driveinterface), or within a processor of the one of access nodes 17, inassociation with one or more streams of data, and terminate at anotherperipheral unit or another processor of the one of access nodes 17. Eachwork unit maintained by a data processing unit is associated with anamount of work that is relevant to the entity executing the work unitfor processing a respective portion of a stream.

Stream processing is typically initiated as a result of receiving one ormore data units associated with respective portions of the stream andconstructing and managing work units for processing respective portionsof the data stream. In protocol processing, a portion would be a singlebuffer (e.g. packet), for example. Within access nodes 17, work unitsmay be executed by processor cores, hardware blocks, I/O interfaces, orother computational processing units. For instance, a processor core ofan access node 17 executes a work unit by accessing the respectiveportion of the stream from memory and performing one or morecomputations in accordance with the work unit. A component of the one ofaccess nodes 17 may receive, execute or generate work units. Asuccession of work units may define how the access node processes aflow, and smaller flows may be stitched together to form larger flows.

For purposes of example, DPUs within each access node 17 may execute anoperating system, such as a general-purpose operating system (e.g.,Linux or other flavor of Unix) or a special-purpose operating system,that provides an execution environment for data plane software for dataprocessing. Moreover, each DPU may be configured to utilize a work unit(WU) stack data structure (referred to as a ‘WU stack’ in a multiplecore processor system. As described herein, the WU stack data structuremay provide certain technical benefits, such as helping manage an eventdriven, run-to-completion programming model of an operating systemexecuted by the multiple core processor system. The WU stack, in a basicform, may be viewed as a stack of continuation WUs used in addition to(not instead of) a program stack maintained by the operating system asan efficient means of enabling program execution to dynamically movebetween cores of the access node while performing high-rate streamprocessing. As described below, a WU data structure is a building blockin the WU stack and can readily be used to compose a processing pipelineand services execution in a multiple core processor system. The WU stackstructure carries state, memory, and other information in auxiliaryvariables external to the program stack for any given processor core. Insome implementations, the WU stack may also provide an exception modelfor handling abnormal events and a ‘success bypass’ to shortcut a longseries of operations. Further, the WU stack may be used as an arbitraryflow execution model for any combination of pipelined or parallelprocessing.

As described herein, access nodes 17 may process WUs through a pluralityof processor cores arranged as processing pipelines within access nodes17, and such processing cores may employ techniques to encourageefficient processing of such work units and high utilization ofprocessing resources. For instance, a processing core (or a processingunit within a core) may, in connection with processing a series of workunits, access data and cache the data into a plurality of segments of alevel 1 cache associated with the processing core. In some examples, aprocessing core may process a work unit and cache data from non-coherentmemory in a segment of the level 1 cache. The processing core may alsoconcurrently prefetch data associated with a work unit expected to beprocessed in the future into another segment of the level 1 cacheassociated with the processing core. By prefetching the data associatedwith the future work unit in advance of the work unit being dequeuedfrom a work unit queue for execution by the core, the processing coremay be able to efficiently and quickly process a work unit once the workunit is dequeued and execution of the work unit is to commence by theprocessing core. More details on work units and stream processing bydata processing units of access nodes are available in U.S. ProvisionalPatent Application No. 62/589,427, filed Nov. 21, 2017, entitled “WorkUnit Stack Data Structures in Multiple Core Processor System,” and U.S.Provisional Patent Application No. 62/625,518, entitled “EFFICIENT WORKUNIT PROCESSING IN A MULTICORE SYSTEM”, filed Feb. 2, 2018, the entirecontents of both being incorporated herein by reference.

As described herein, the data processing units for access nodes 17includes one or more specialized hardware-based accelerators configuredto perform acceleration for various data-processing functions, therebyoffloading tasks from the processing units when processing work units.That is, each accelerator is programmable by the processing cores, andone or more accelerators may be logically chained together to operate onstream data units, such as by providing cryptographic functions,compression and regular expression (RegEx) processing, data durabilityfunctions, data storage functions and networking operations.

In FIG. 1, and in accordance with one or more aspects of the presentdisclosure, one or more of access nodes 17 may store data across datacenter 10. For instance, in the example of FIG. 1, access node 17-1 ofNCSU 40-1 receives data from one or more servers 12 (e.g., one ofservers 12-1 through 12-N). Access node 17-1 processes the data byapplying a data durability encoding algorithm. In some examples, thedata durability encoding algorithm is a Reed Solomon erasure codingalgorithm that results in the data being split into multiple datafragments. In addition, the multiple data fragments are used to generateone or more parity data fragments. Performing an erasure codingoperation may, in some examples, require significant computingresources. If access node 17-1 performs the erasure coding operation,however, each of servers 12 may be able to perform other productiveoperations.

Access node 17-1 stores each of the data fragments (both the datafragments resulting from the split and also the parity data fragments)across data center 10. In the example of FIG. 1, access node 17-1 storeseach of the data fragments within a different fault domain, where eachfault domain corresponds to each of NCSUs 40. In other words, each datafragment generated by access node 17-1 is stored within a different NCSU40 across data center 10. If the failure of one NCSU 40 is unlikely tobe correlated with the failure of another NCSU 40, storing datafragments in different NCSUs 40 may reduce the likelihood that more thanone data fragment will be lost or unavailable due to a hardware orsoftware error (or other cause of unavailability).

After storing the data fragments, access node 17-1 may receive a request(e.g., a “read” request) for a portion of or all of the stored data thathas stored across data center 10 as data fragments. For instance, in theexample of FIG. 1, access node 17-1 may receive a request, from one ofservers 12, to read the same data that access node 17 stored as datafragments across data center 10, as described above. After receiving therequest, access node 17-1 accesses a directory, stored within accessnode 17-1 or elsewhere, identifying where each of the data fragments arestored within data center 10. Access node 17 determines whether each ofthe data fragments are available at the location specified in thedirectory. If the data fragments are available, access node 17-1retrieves the data fragments from each specified location (i.e., eachfailure domain) and reassembles them into the original data.

If one or more of the data fragments are not available, however, accessnode 17-1 accesses one or more of the parity data fragments and uses theparity data fragments, along with the available data fragments, toreconstruct the original data. To do so, access node 17-1 performs adata durability decoding operation to reconstruct the data. If the datawas encoded using a Reed Solomon erasure coding algorithm, for example,the decoding operation involves a corresponding Reed Solomon decodingoperation. As with the encoding operation, the decoding operation may bea computationally intensive operation. However, if access node 17-1performs some or all of the processing associated with the decodingoperation, servers 12 may again continue other productive operations,and at least some of the latency and/or expenditure of servercomputational resources that might otherwise be associated with theerasure coding operations is avoided. When the decode operation iscomplete, the reconstructed data is output to the requesting server 12as a response to the read request.

Through techniques in accordance with one or more aspects of the presentdisclosure, such as by storing data in fragments across multiple failuredomains in a manner that enables efficient recovery of the data usingonly a subset of the data, data center 10 may perform inline recovery ofactively used data. By performing inline recovery of actively-used orhot data, data center 10 may operate reliably and efficiently becausedata will be available and quickly accessible, even in situations wherea hardware or software error (or other event causing aspects of anetwork to be impacted) occurs. Therefore, aspects of this disclosuremay improve the function of data center 10 because data durabilitycoding on a network, in a manner consistent with techniques describedherein, may have the effect of causing data center 10 to be morereliable and efficient.

Further, by storing data across multiple fault domains in a manner thatenables efficient recovery of the data using only a subset of the data,it may be possible to use data durability techniques that require lessstorage, since multiple independent fault domains are less likely tofail at the same time. By using data durability techniques that requireless storage, data center 10 may effectively be able to store more data,since the data durability techniques would not require as much redundantstorage of data, thereby effectively increasing the space that can beused for non-redundant storage of data. Therefore, aspects of thisdisclosure may improve the function of data center 10 because datadurability coding on a network, in a manner consistent with thetechniques described herein, may have the effect of increasing effectivestorage capacity of data center 10.

In some examples, some or all of access nodes 17 may be configured toperform data durability operations in a flexible manner, implementing arange of data durability methods and functions from simple to complex.Such schemes may range from data replication schemes, simple parityencoding schemes, RAID encoding schemes, erasure coding schemes (e.g.,Reed Solomon encoding schemes), and hierarchical erasure coding schemes.To implement some or all of such schemes, each of access nodes 17 may beconfigured to implement a common and/or unified matrix approach thatimplements different data durability schemes based on a matrix ofcoefficients chosen or configured for each scheme. By choosing orselecting a different matrix of coefficients, each of access nodes 17may perform or implement a different approach to data durability.

Through techniques in accordance with one or more aspects of the presentdisclosure, such as by implementing each of a variety of durabilitycoding and/or erasure coding schemes using a common matrix approach, oneor more of access nodes 17 may provide an abstraction layer for datadurability that enables flexible configuration and softwareprogrammability. In such an implementation, each of access nodes 17 mayoperate more reliably and efficiently, since software developed foraccess nodes 17 may be based on the abstraction layer, thereby reducingthe complexity required to develop and test software for implementingdata durability functions. Therefore, aspects of this disclosure mayimprove the function of access node 17 because implementing durabilitycoding and/or erasure coding schemes using a common matrix approach mayhave the effect of causing software developed for access node 17 to beof higher quality, thereby enabling access node 17 to operate morereliably and efficiently.

FIG. 2 is a block diagram illustrating an example data processing unit(DPU) 130 including two or more processing cores, in accordance with thetechniques of this disclosure. DPU 130 generally represents a hardwarechip implemented in digital logic circuitry and may be used in anycomputing or network device. DPU 130 may operate substantially similarto and generally represent any of access nodes 17 of FIG. 1. Thus, DPU130 may be communicatively coupled to one or more network devices,server devices (e.g., servers 12), random access memory, storage media(e.g., solid state drives (SSDs)), a data center fabric (e.g., switchfabric 14), or the like, e.g., via PCI-e, Ethernet (wired or wireless),or other such communication media. Moreover, DPU 130 may be implementedas one or more application-specific integrated circuit (ASIC), may beconfigurable to operate as a component of a network appliance or may beintegrated with other DPUs within a device.

In the illustrated example of FIG. 2, DPU 130 includes a multi-coreprocessor 132 having a plurality of programmable processing cores140A-140N (“cores 140”) coupled to an on-chip memory unit 134. Each ofcores 140 includes a level 1 cache 141 (level 1 caches 141A, 141B, and141N are associated with cores 140A, 140B, and 140N, respectively).

Memory unit 134 may include two types of memory or memory devices,namely coherent cache memory 136 and non-coherent buffer memory 138.Processor 132 also includes a networking unit 142, work unit (WU) queues143, a memory controller 144, and accelerators 146. As illustrated inFIG. 2, each of cores 140, networking unit 142, WU queues 143, memorycontroller 144, memory unit 134, and accelerators 146 arecommunicatively coupled to each other. In some examples, processor 132of DPU 130 further includes one or more accelerators (not shown)configured to perform acceleration for various data-processingfunctions, such as look-ups, matrix multiplication, cryptography,compression, regular expressions, or the like.

In this example, DPU 130 represents a high performance, hyper-convergednetwork, storage, and data processor and input/output hub. For example,networking unit 142 may be configured to receive one or more datapackets from and transmit one or more data packets to one or moreexternal devices, e.g., network devices. Networking unit 142 may performnetwork interface card functionality, packet switching, and the like,and may use large forwarding tables and offer programmability.Networking unit 142 may expose Ethernet ports for connectivity to anetwork, such as switch fabric 14 of FIG. 1. DPU 130 may also includeone or more interfaces for connectivity to host devices (e.g., servers)and data storage devices, e.g., solid state drives (SSDs) via PCIelanes. DPU 130 may further include one or more high bandwidth interfacesfor connectivity to off-chip external memory.

Processor 132 further includes accelerators 146 configured to performacceleration for various data-processing functions, such as look-ups,matrix multiplication, cryptography, compression, data durability and/orreliability, regular expressions, or the like. For example, accelerators146 may comprise hardware implementations of look-up engines, matrixmultipliers, cryptographic engines, compression engines, or the like. Inaccordance with the techniques of this disclosure, at least one ofaccelerators 146 represents a data durability unit that may be used toimplement one or more data durability and/or reliability schemes. Insome examples, such a data durability unit may be configured to performmatrix multiplication operations commonly performed in erasure codingschemes, such as Reed Solomon erasure coding schemes. Such a datadurability unit may be configured to efficiently perform operations,such as those relating to Galois Field mathematics, that might bedifficult and/or inefficient to perform using commonly availableprocessors or other processing hardware. Further, such a data durabilityunit may be designed to perform and/or implement multiple differenttypes of data durability schemes by configuring different matricesspecific to each implementation. In such an example, implementing adifferent data durability scheme may involve choosing or generating adifferent coefficient matrix for use in encoding and/or decoding data,as further discussed below.

Memory controller 144 may control access to on-chip memory unit 134 bycores 140, networking unit 142, and any number of external devices,e.g., network devices, servers, external storage devices, or the like.Memory controller 144 may be configured to perform a number ofoperations to perform memory management in accordance with the presentdisclosure. For example, memory controller 144 may be capable of mappingaccesses from one of the cores 140 to either of coherent cache memory136 or non-coherent buffer memory 138. More details on the bifurcatedmemory system included in the DPU are available in U.S. ProvisionalPatent Application No. 62/483,844, filed Apr. 10, 2017, and titled“Relay Consistent Memory Management in a Multiple Processor System,” theentire content of which is incorporated herein by reference.

Cores 140 may comprise one or more microprocessors without interlockedpipeline stages (MIPS) cores, advanced reduced instruction set computing(RISC) machine (ARM) cores, performance optimization with enhancedRISC-performance computing (PowerPC) cores, RISC Five (RISC-V) cores, orcomplex instruction set computing (CISC or x86) cores. Each of cores 140may be programmed to process one or more events or activities related toa given data packet such as, for example, a networking packet or astorage packet. Each of cores 140 may be programmable using a high-levelprogramming language, e.g., C, C++, or the like.

Each of level 1 caches 141 may include a plurality of cache lineslogically or physically divided into cache segments. Each of level 1caches 141 may be controlled by a load/store unit also included withinthe core. The load/store unit may include logic for loading data intocache segments and/or cache lines from non-coherent buffer memory 138and/or memory external to DPU 130. The load/store unit may also includelogic for flushing cache segments and/or cache lines to non-coherentbuffer memory 138 and/or memory external to DPU 130. In some examples,the load/store unit may be configured to prefetch data from main memoryduring or after a cache segment or cache line is flushed.

As described herein, processor cores 140 may be arranged as processingpipelines, and such processing cores may employ techniques to encourageefficient processing of such work units and high utilization ofprocessing resources. For instance, any of processing cores 140 (or aprocessing unit within a core) may, in connection with processing aseries of work units retrieved from WU queues 143, access data and cachethe data into a plurality of segments of level 1 cache 141 associatedwith the processing core. In some examples, a processing core 140 mayprocess a work unit and cache data from non-coherent memory 138 in asegment of the level 1 cache 141. As described herein, concurrent withexecution of work units by cores 140, a load store unit of memorycontroller 144 may be configured to prefetch, from non-coherent memory138, data associated with work units within WU queues 143 that areexpected to be processed in the future, e.g., the WUs now at the top ofthe WU queues and next in line to be processed. For each core 140, theload store unit of memory controller 144 may store the prefetched dataassociated with the WU to be processed by the core into a standbysegment of the level 1 cache 141 associated with the processing core140.

In some examples, the plurality of cores 140 executes instructions forprocessing a plurality of events related to each data packet of one ormore data packets, received by networking unit 142, in a sequentialmanner in accordance with one or more work units associated with thedata packets. As described above, work units are sets of data exchangedbetween cores 140 and networking unit 142 where each work unit mayrepresent one or more of the events related to a given data packet.

As one example use case, stream processing may be divided into workunits executed at a number of intermediate processors between source anddestination. Depending on the amount of work to be performed at eachstage, the number and type of intermediate processors that are involvedmay vary. In processing a plurality of events related to each datapacket, a first one of the plurality of cores 140, e.g., core 140A mayprocess a first event of the plurality of events. Moreover, first core140A may provide to a second one of plurality of cores 140, e.g., core140B a first work unit of the one or more work units. Furthermore,second core 140B may process a second event of the plurality of eventsin response to receiving the first work unit from first core 140B.

As another example use case, transfer of ownership of a memory bufferbetween processing cores may be mediated by a work unit messagedelivered to one or more of processing cores 140. For example, the workunit message may be a four-word message including a pointer to a memorybuffer. The first word may be a header containing information necessaryfor message delivery and information used for work unit execution, suchas a pointer to a function for execution by a specified one ofprocessing cores 140. Other words in the work unit message may containparameters to be passed to the function call, such as pointers to datain memory, parameter values, or other information used in executing thework unit.

In one example, receiving a work unit is signaled by receiving a messagein a work unit receive queue (e.g., one of WU queues 143). The one of WUqueues 143 is associated with a processing element, such as one of cores140, and is addressable in the header of the work unit message. One ofcores 140 may generate a work unit message by executing storedinstructions to addresses mapped to a work unit transmit queue (e.g.,another one of WU queues 143). The stored instructions write thecontents of the message to the queue. The release of a work unit messagemay be interlocked with (gated by) flushing of the core's dirty cachedata and in some examples, prefetching into the cache of data associatedwith another work unit for future processing.

FIG. 3 is a block diagram illustrating another example of a DPU 150including two or more processing clusters, in accordance with thetechniques of this disclosure. DPU 150 may operate substantially similarto any of the access nodes 17 of FIG. 1. Thus, DPU 150 may becommunicatively coupled to a data center fabric (e.g., switch fabric14), one or more server devices (e.g., servers 12), storage media (e.g.,SSDs), one or more network devices, random access memory, or the like,e.g., via PCI-e, Ethernet (wired or wireless), or other suchcommunication media in order to interconnect each of these variouselements. DPU 150 generally represents a hardware chip implemented indigital logic circuitry. As various examples, DPU 150 may be provided asan integrated circuit mounted on a motherboard of a computing,networking and/or storage device or installed on a card connected to themotherboard of the device.

In general, DPU 150 represents a high performance, hyper-convergednetwork, storage, and data processor and input/output hub. Asillustrated in FIG. 3, DPU 150 includes networking unit 152, processingclusters 156A-1 to 156N-M (processing clusters 156), host units 154A-1to 154B-M (host units 154), and central cluster 158, and is coupled toexternal memory 170. Each of host units 154, processing clusters 156,central cluster 158, and networking unit 152 may include a plurality ofprocessing cores, e.g., MIPS cores, ARM cores, PowerPC cores, RISC-Vcores, or CISC or x86 cores. External memory 170 may comprise randomaccess memory (RAM) or dynamic random access memory (DRAM).

As shown in FIG. 3, host units 154, processing clusters 156, centralcluster 158, networking unit 152, and external memory 170 arecommunicatively interconnected via one or more specializednetwork-on-chip fabrics. A set of direct links 162 (represented asdashed lines in FIG. 3) forms a signaling network fabric that directlyconnects central cluster 158 to each of the other components of DPU 150,that is, host units 154, processing clusters 156, networking unit 152,and external memory 170. A set of grid links 160 (represented as solidlines in FIG. 3) forms a data network fabric that connects neighboringcomponents (including host units 154, processing clusters 156,networking unit 152, and external memory 170) to each other in atwo-dimensional grid.

Networking unit 152 has Ethernet interfaces 164 to connect to the switchfabric, and interfaces to the data network formed by grid links 160 andthe signaling network formed by direct links 162. Networking unit 152provides a Layer 3 (i.e., OSI networking model Layer 3) switchforwarding path, as well as network interface card (NIC) assistance. Oneor more hardware direct memory access (DMA) engine instances (not shown)may be attached to the data network ports of networking unit 152, whichare coupled to respective grid links 160. The DMA engines of networkingunit 152 are configured to fetch packet data for transmission. Thepacket data may be in on-chip or off-chip buffer memory (e.g., withinbuffer memory of one of processing clusters 156 or external memory 170),or in host memory.

Host units 154 each have PCI-e interfaces 166 to connect to serversand/or storage devices, such as SSD devices. This allows DPU 150 tooperate as an endpoint or as a root. For example, DPU 150 may connect toa host system (e.g., a server) as an endpoint device, and DPU 150 mayconnect as a root to endpoint devices (e.g., SSD devices). Each of hostunits 154 may also include a respective hardware DMA engine (not shown).Each DMA engine is configured to fetch data and buffer descriptors fromhost memory, and to deliver data and completions to host memory.

DPU 150 provides optimizations for stream processing. DPU 150 executesan operating system that facilitates run-to-completion processing, whichmay eliminate interrupts, thread scheduling, cache thrashing, andassociated costs. For example, an operating system may run on one ormore of processing clusters 156. Central cluster 158 may be configureddifferently from processing clusters 156, which may be referred to asstream processing clusters. In one example, central cluster 158 executesthe operating system kernel (e.g., Linux kernel) as a control plane.Processing clusters 156 may function in run-to-completion thread mode ofa data plane software stack of the operating system. That is, processingclusters 156 may operate in a tight loop fed by work unit queuesassociated with each processing core in a cooperative multi-taskingfashion.

DPU 150 operates on work units (WUs) that associate a buffer with aninstruction stream to reduce dispatching overhead and allow processingby reference to minimize data movement and copy. The stream-processingmodel may structure access by multiple processors (e.g., processingclusters 156) to the same data and resources, avoid simultaneoussharing, and therefore, reduce contention. A processor may relinquishcontrol of data referenced by a work unit as the work unit is passed tothe next processor in line. Central cluster 158 may include a centraldispatch unit responsible for work unit queuing and flow control, workunit and completion notification dispatch, and load balancing andprocessor selection from among processing cores of processing clusters156 and/or central cluster 158.

As described above, work units are sets of data exchanged betweenprocessing clusters 156, networking unit 152, host units 154, centralcluster 158, and external memory 170. Each work unit may be representedby a fixed length data structure, or message, including an action valueand one or more arguments. In one example, a work unit message includesfour words, a first word having a value representing an action value andthree additional words each representing an argument. The action valuemay be considered a work unit message header containing informationnecessary for message delivery and information used for work unitexecution, such as a work unit handler identifier, and source anddestination identifiers of the work unit. The other arguments of thework unit data structure may include a frame argument having a valueacting as a pointer to a continuation work unit to invoke a subsequentwork unit handler, a flow argument having a value acting as a pointer tostate that is relevant to the work unit handler, and a packet argumenthaving a value acting as a packet pointer for packet and/or blockprocessing handlers.

In some examples, one or more processing cores of processing clusters180 may be configured to execute program instructions using a work unit(WU) stack. In general, a work unit (WU) stack is a data structure tohelp manage event driven, run-to-completion programming model of anoperating system typically executed by processing clusters 156 of DPU150, as further described in U.S. Patent Application Ser. No.62/589,427, filed Nov. 21, 2017, the entire content of which isincorporated herein by reference.

As described herein, in some example implementations, load store unitswithin processing clusters 156 may, concurrent with execution of workunits by cores within the processing clusters, identify work units thatare enqueued in WU queues for future processing by the cores. In someexamples, WU queues storing work units enqueued for processing by thecores within processing clusters 156 may be maintained as hardwarequeues centrally managed by central cluster 158. In such examples, loadstore units may interact with central cluster 158 to identify futurework units to be executed by the cores within the processing clusters.The load store units prefetch, from the non-coherent memory portion ofexternal memory 170, data associated with the future work units. Foreach core within processing clusters 156, the load store units of thecore may store the prefetched data associated with the WU to beprocessed by the core into a standby segment of the level 1 cacheassociated with the processing core.

FIG. 4 is a block diagram illustrating an example processing cluster 180including two or more processing cores 182A-182N. Each of processingclusters 156 of DPU 150 of FIG. 3 may be configured in a mannersubstantially similar to processing cluster 180 shown in FIG. 4. In theexample of FIG. 4, processing cluster 180 includes cores 182A-182N(“cores 182”), a memory unit 183 including a coherent cache memory 184and a non-coherent buffer memory 186, a cluster manager 185 including WUqueue manager 187 for maintaining (e.g., within hardware registers ofprocessing cluster 180) and manipulating WU queues 188, and accelerators189A-189X (“accelerators 189”). Each of cores 182 includes L1 buffercache 198 (i.e., core 182 includes L1 buffer cache 198A and in general,core 182N includes L1 buffer cache 198N). In some examples, clustermanager 185 is alternatively located within central cluster 158, and/orWU queues 188 are alternatively maintained within central cluster 158(e.g., within hardware registers of central cluster 158).

An access node or DPU (such as access nodes 17 of FIG. 1, DPU 130 ofFIG. 2, or DPU 150 of FIG. 3) may support two distinct memory systems: acoherent memory system and a non-coherent buffer memory system. In theexample of FIG. 4, coherent cache memory 184 represents part of thecoherent memory system while non-coherent buffer memory 186 representspart of the non-coherent buffer memory system. Cores 182 may representthe processing cores discussed with respect to DPU 150 of FIG. 3. Cores182 may share non-coherent buffer memory 186. As one example, cores 182may use non-coherent buffer memory 186 for sharing streaming data, suchas network packets.

In general, accelerators 189 perform acceleration for variousdata-processing functions, such as table lookups, matrix multiplication,cryptography, compression, data durability, regular expressions, or thelike. That is, accelerators 189 may comprise hardware implementations oflookup engines, matrix multipliers, cryptographic engines, compressionengines, data durability encoders and/or decoders, regular expressioninterpreters, or the like. For example, accelerators 189 may include amatrix multiplication engine, or a lookup engine that performs hashtable lookups in hardware to provide a high lookup rate. A lookupengine, for example, may be invoked through work units from externalinterfaces and virtual processors of cores 182, and generates lookupnotifications through work units. Accelerators 189 may also include oneor more cryptographic units to support various cryptographic processes.Accelerators 189 may also include one or more compression units toperform compression and/or decompression. Accelerators 189 may furtherinclude one or more data durability units to perform functions relatingto data durability, erasure coding, and/or data reliability.

An example process by which a processing cluster 180 processes a workunit is described here. Initially, cluster manager 185 of processingcluster 180 may queue a work unit (WU) in a hardware queue of WU queues188. When cluster manager 185 “pops” the work unit from the hardwarequeue of WU queues 188, cluster manager 185 delivers the work unit toone of accelerators 189, e.g., a lookup engine. The accelerator 189 towhich the work unit is delivered processes the work unit and determinesthat the work unit is to be delivered to one of cores 182 (inparticular, core 182A, in this example) of processing cluster 180. Thus,the one of accelerators 189 forwards the work unit to a local switch ofthe signaling network on the DPU, which forwards the work unit to bequeued in a virtual processor queue of WU queues 188.

As noted above, in accordance with the techniques of this disclosure,one or more of accelerators 189 may be configured to data durabilityfunctions and/or erasure coding functions. A data durability acceleratorof accelerators 189, in accordance with the techniques of thisdisclosure, may include processing circuitry capable of efficientlyperforming erasure coding operations, which may, in some examples,involve matrix multiplication and/or Galois Field mathematicaloperations. Such a data durability accelerator may enable storage ofdata fragments, including parity data fragments, across different faultdomains within data center 10. Further, such a data durabilityaccelerator may enable retrieval and reconstruction of data where only asubset of the original data fragments are available within data center10.

After cluster manager 185 pops the work unit from the virtual processorqueue of WU queues 188, cluster manager 185 delivers the work unit via acore interface to core 182A, in this example. An interface unit of core182A then delivers the work unit to one of the virtual processors ofcore 182A.

Core 182A processes the work unit, which may involve accessing data,such as a network packet or storage packet, in non-coherent memory 156Aand/or external memory 170. Core 182A may first look for thecorresponding data in cache 198A, and in the event of a cache miss, mayaccess the data from non-coherent memory 156A and/or external memory170. In some examples, while processing the work unit, core 182A maystore information (i.e., the network packet or data packet) associatedwith the work unit in an active segment of cache 198A. Further, core182A may, while processing the work unit, prefetch data associated witha second work unit into a different, standby segment of cache 198A. Whencore 182A completes processing of the work unit, core 182A initiates (orcauses initiation of) a cache flush for the active segment, and may alsoinitiate prefetching of data associated with a third work unit (to beprocessed later) into that active segment. Core 182A (or a virtualprocessor within core 182A) may then swap the active segment and thestandby segment so that the previous standby segment becomes the activesegment for processing of the next work unit (i.e., the second workunit). Because data associated with the second work unit was prefetchedinto this now active segment, core 182A (or a virtual processor withincore 182A) may be able to more efficiently process the second work unit.Core 182A then outputs corresponding results (possibly including one ormore work unit messages) from performance of the work unit back throughthe interface unit of core 182A.

As described herein, in some example implementations, load store unitswithin memory unit 183 may, concurrent with execution of work units bycores 182 within the processing cluster 180, identify work units thatare enqueued in WU queues 188 for future processing by the cores. Theload store units prefetch, from a non-coherent memory portion ofexternal memory 170, data associated with the future work units andstore the prefetched data associated with the WUs to be processed by thecores into a standby segment of the level 1 cache associated with theparticular processing cores.

FIG. 5 is a block diagram illustrating an example accelerator, inaccordance with one or more aspects of the present disclosure. In theexample of FIG. 5, accelerator 200 may be one of accelerators 146 ofdata processing unit 130 from FIG. 2, or one of accelerators 189 ofprocessing cluster 180 from FIG. 4. In some examples, accelerator 200may be included in some or all of processing clusters 156 of dataprocessing unit 150 illustrated in FIG. 3.

In the example of FIG. 5, accelerator 200 is configured to accelerate,improve, and/or modify operations relating to data durability and/orreliability that might otherwise be performed by software executing on ageneral purpose processor. As illustrated in FIG. 5, accelerator 200 mayinclude one or more gather blocks 202, one or more data durabilityblocks 206, one or more security blocks 208, and one or more scatterblocks 212. Further, in some examples, other types of specific-functionblocks, beyond data durability block 206 and security block 208, mayalso be included within accelerator 200. In addition, as illustrated inFIG. 5, pass-through connection 209 may also be included withinaccelerator 200. Data durability block 206 and security block 208 mayeach be implemented as a DMA inline accelerator positioned betweengather block 202 and scatter block 212. For data durability block 206,gather block 202 may read a coefficient matrix and data fragmentsthrough gather commands, and scatter block 212 may write data fragmentsand/or parity fragments back to system memory through scatter softwarecommands. Accordingly, gather block 202 may provide data accessed froman external memory, and may serve as an ingress DMA device. Scatterblock 212 may send data back to external memory, and may serve as anegress DMA device.

Through these components and/or others described herein, accelerator 200may support multiple different data durability or erasure coding schemes(e.g., through data durability block 206), enabling data to be reliablystored and retrieved from locations within data center 10. Accelerator200 may also support security functions (e.g., through security block208), enabling data received from gather block 202 to be encryptedand/or decrypted before being provided to scatter block 212.

In FIG. 5, and in accordance with one or more aspects of the presentdisclosure, accelerator 200 may, pursuant to an erasure codingalgorithm, encode data stored within processing cluster 180. Forinstance, in the example of FIG. 5 and with reference to FIG. 4, gatherblock 202 receives and gathers multiple streams of data from memory unit183 within processing cluster 180. Gather block 202 outputs the gatheredstreams to data durability block 206. Data durability block 206 performsan erasure code encoding operation, splitting data into data fragmentsand generating one or more parity fragments. Data durability block 206outputs the data fragments to scatter block 212. Scatter block 212scatters the fragments across memory unit 183 within processing cluster180. In some examples, data durability block 206 may perform erasurecoding operations based on portions of data fragments pursuant to astrided read operation as further described herein (e.g., see FIG. 8C).

FIG. 6A is a conceptual diagram illustrating an example write operationperformed pursuant to a basic replication data reliability scheme, inaccordance with one or more aspects of the present disclosure. FIG. 6Aillustrates a number of racks 70, each connected through switch fabric14. In the example of FIG. 6A, each of racks 70 may be considered aseparate failure domain. Each of racks 70 include one or more accessnodes 17 and one or more servers 12. For example, in FIG. 6A, rack 70-1includes access node 17 and server 12, which may correspond to one ofaccess nodes 17 within one of servers 12 of FIG. 1. Further, in someexamples, each of access nodes 17 may correspond to data processing unit130 as illustrated in FIG. 2 or data processing unit 150 as illustratedin FIG. 3.

Also as illustrated in FIG. 6A, access node 17 of rack 70-1 includes oneor more accelerators 200. Accelerator 200 includes one or more datadurability blocks 206, which may correspond to data durability block 206as illustrated in FIG. 5.

In the example of FIG. 6A, and in accordance with one or more aspects ofthe present disclosure, access node 17 may store duplicate copies ofdata 801 across network 8. For instance, in the example of FIG. 6A,access node 17 receives data 801 from server 12. Access node 17 alsoreceives a command to store data 801. In response to the command, accessnode 17 outputs data 801 to accelerator 200. Accelerator 200 passes data801 through data durability block 206, which replicates data 801. Insome examples, data durability block 206 replicates data 801 a number oftimes sufficient to ensure a desired level of reliability for storage ofdata 801. In some examples, data durability block 206 may replicate data801 four times as illustrated in the example of FIG. 6A. In otherexamples, the data may be replicated more times (e.g., six times) orfewer times. After replicating data 801, access node 17 outputs each ofthe replicated copies over switch fabric 14, and each replicated copy ofthe data set is stored within a different one of racks 70 across network8. If one of data sets 801 fails or is unavailable, access node 17 mayrecover the data by reading from any of the other data sets 801 storedon the network.

In the example of FIG. 6A, each of racks 70 is considered an independentfailure domain, so access node 17 may store each of the replicated datasets 801 within a different rack 70. In other examples, however, it maybe desirable, necessary, or otherwise appropriate for some of replicateddata sets 801 may be stored within the same rack.

As illustrated in FIG. 6A, data durability block 206 within rack 70-1stores data within network 8 pursuant to a data durability scheme thatinvolves simply replicating data 801, and storing replicas of data 801within failure domains across network 8. This type of data durabilityscheme tends to exhibit fast recovery times, little complexity, andlittle or no computational resource requirements. Further, the exampleof FIG. 6A has low read costs because to recover from a dataunavailability event impacting data 801, access node 17 might simplyread from one of the replicated data sets 801 stored across network 8.Further, access node 17 might only have to read from one of theduplicate data sets 801 a single time. However, for the durabilityscheme illustrated in FIG. 6A, storage costs are high—more than 100%storage overhead—since a significant amount of storage is required tostore duplicate copies of data 801.

FIG. 6B is a conceptual diagram illustrating an example write operationperformed pursuant to a simple parity encoding data reliability scheme,in accordance with one or more aspects of the present disclosure. As inFIG. 6A, FIG. 6B includes a number of racks 70, each connected throughswitch fabric 14. Rack 70-1 includes one or more access nodes 17 and oneor more servers 12. Access node 17 includes one or more accelerators 200and one or more data durability blocks 206. In FIG. 6B, data durabilityblock 206 generates data fragment 802P from data fragments 802D-1,802D-2, 802D-3, 802D-4, and 802D-5 (collectively “data fragments 802D”).In this example, data fragment 802P serves as a parity block that may beused to recover any of data fragments 802D that may become unavailable.

In the example of FIG. 6B, an in accordance with one or more aspects ofthe present disclosure, access node 17 may store data fragments derivedfrom data 801 across network 8. For instance, in the example of FIG. 6A,access node 17 receives data 801 from server 12. Access node 17 outputsdata 801 to accelerator 200. Accelerator 200 passes data 801 throughdata durability block 206, which splits data 801 into five equal-sizedor approximately equal-sized data fragments 802. In other examples, datadurability block 206 may split data 801 into a different number offragments. Data durability block 206 creates data fragment 802P byapplying a simple parity encoding algorithm.

In some examples, data durability block 206 may, to implement the parityencoding algorithm, generate data fragment 802P based on a simpleodd/even parity scheme. For instance, to generate data fragment 802Pfrom data fragments 802D, and in a simplified example involvingequal-sized data fragments 802D, one bit from each of data fragments802D is used to form a collection of bits. A parity or check bit for thecollection of bits is generated based on whether the total number of “1”bits in the collection of bits is odd or even. For instance, for an oddparity scheme, if the number of “1” bits in the collection of bits iseven, data durability block 206 generates a “1” parity bit, and if thenumber of “1” bits is odd, data durability block 206 generates a “0”parity bit. Data durability block 206 continues this process for eachrespective bit in each of the equal-sized data fragments 802D. Datadurability block 206 combines the resulting check bits generated fromthe collection of bits from data fragments 802D to form data fragment802P, also having a size equal to the size of each of data fragments802D.

An even parity scheme would operate in a manner similar to the oddparity scheme example described above. For an even parity scheme, datadurability block 206 may generate a “0” parity bit for each collectionof bits from data fragments 802D that has an even number of “1” bits,and may generate a “1” parity bit for each collection of bits having anodd number of “1” bits. In such an example, data durability block 206combines the resulting check bits to form the corresponding datafragment 802P.

Once data fragment 802P is encoded and stored pursuant to either an evenor odd parity scheme, if one of data fragments 802D is lost, datadurability block 206 may use the parity bits in data fragment 802P(along with an indication that the parity bits in data fragment 802Dwere generated pursuant to an odd parity scheme) to reconstruct the lostdata fragment 802D. As a result, data durability block 206 may becapable of recovering any one of data fragments 802D if data fragment802P is available.

After data durability block 206 creates data fragment 802P, access node17 outputs data fragment 802P and each of data fragments 802D overswitch fabric 14, and stores each data fragment within a different faultdomain (e.g., different racks 70) across network 8. If one of datafragments 802D later fails or becomes unavailable, access node 17 may,in order to access data 801, recover the unavailable data fragment 802Dby reading from the remaining available data fragments 802D and alsodata fragment 802P. Data durability block 206 generates the missing datafragment 802D from the remaining data fragments 802D and data fragment802P pursuant to the parity decoding procedures described above.

The simple parity data durability scheme illustrated in FIG. 6B anddescribed above requires very low storage resources, since only a smallamount of storage overhead is required (in the example of FIG. 6B, theadditional storage overhead is ⅕ or 20%). Further, the computingcomplexity is relatively low, and may require only exclusive-oroperations and/or binary addition operations. However, the system ofFIG. 6B typically will only tolerate loss of one of data fragments 802D.In other words, if one data fragment (any of data fragment 802P and datafragments 802) is unavailable, the missing data fragment can bereconstructed from the remaining data fragments. However, if more thanone data fragment is lost, the system of FIG. 6B might be unable tofully reconstruct the original data 801.

FIG. 6C is a conceptual diagram illustrating an example write operationperformed pursuant to an erasure coding data reliability scheme, inaccordance with one or more aspects of the present disclosure. As inFIG. 6A and FIG. 6B, FIG. 6C includes a number of racks 70 connectedthrough switch fabric 14, each of racks 70 including one or more accessnodes 17 and one or more servers 12. In FIG. 6C, data durability block206 is configured to apply an erasure coding approach to datadurability. Erasure coding is a method of data protection in which datais broken into fragments, expanded and encoded with redundant datapieces. Erasure codes enable data fragments that become corrupted atsome point in the storage process to be reconstructed by usinginformation about the data that is stored elsewhere. If some combinationof one or more of the data fragments are erased or are otherwiseunavailable, but the remaining data blocks are available, the erased orunavailable data blocks can be reconstructed from the available blocks.

In some examples, an erasure coding algorithm splits data blocks into“d” data blocks and “p” parity blocks. A Reed Solomon 4+2 erasure codingscheme, for example, uses d=4 data blocks to generate p=2 parity blocks.Many other Reed Solomon implementations are possible, including 12+3,10+4, 8+2, and 6+3 schemes. Other types of erasure encoding schemesbeyond Reed Solomon schemes include parity array codes (e.g., EvenOddcodes, X codes, HoVer codes, WEAVER codes), Low Density Parity Check(LDPC) codes, or Local Reconstruction Codes (LRC). In some cases, suchfor parity array codes, reliability schemes may be more restrictive interms of an ability to recover from failure for a given set ofunavailable data fragments or data blocks. Further, data recovery forparity array codes may be iterative if more than one data fragment ordata block is unavailable; such iterative data recovery may involvetime-consuming and/or inefficient processing, thereby leading to latencyand/or poor performance.

In the example of FIG. 6C, an in accordance with one or more aspects ofthe present disclosure, access node 17 may store, across network 8, datafragments generated pursuant to an erasure encoding scheme. Forinstance, in the example of FIG. 6C, access node 17 receives data 801from server 12. Access node 17 outputs data 801 to accelerator 200.Accelerator 200 feeds data 801 through data durability block 206, whichsplits data 801 into data fragments 802D-1, 802D-2, 802D-3, 802D-4, and802D-5 (collectively “data fragments 802D”). Data durability block 206applies an erasure coding encoding algorithm to generate data fragment802P-1 and data fragment 802P-2 (collectively “data fragments 802P”)from data fragments 802D. Access node 17 stores data fragments 802D anddata fragments 802P within different failure domains (e.g., racks 70)across network 8. If one or more of data fragments 802D or datafragments 802P become unavailable, access node 17 may recover data 801by reading from the remaining available data fragments (which may be acombination of data fragments 802D and/or data fragments 802P) stored onthe network.

The erasure coding scheme illustrated in FIG. 6C involves relatively lowstorage costs, since only a small amount of storage overhead is required(in the example of FIG. 6B, the additional storage overhead is 2/4 or50%). Further, the system of FIG. 6B will tolerate more than the loss ofone data fragment. For example, a Reed Solomon 4+2 erasure coding schemegenerates two parity blocks (p=2) for each set of four data blocks(d=4). And in general, such an erasure coding scheme can correct anumber of erasures less than or equal to “p.” So for the 4+2 system ofFIG. 6C, the loss or unavailability of any two of data fragments (anycombination of data fragment 802D and/or data fragments 802P) stillenables data 801 to be fully recovered by decoding the remaining datafragments. Unavailability of any three data fragments, however, mayresult in data loss. Similarly, for a 12+3 Reed Solomon system,unavailability of any three data or parity fragments still enables theoriginal data to be reconstructed from the remaining fragments, butunavailability of four or more fragments may result in data loss.

One of the drawbacks of erasure coding systems is complexity, andencoding and decoding data using an erasure coding scheme may requirehigh computing resources, complexity, and/or costs. For example, a ReedSolomon erasure coding scheme is typically implemented using GaloisField mathematics, and many current processors are not well equipped toperform Galois Field mathematics operations efficiently. Complexity,computing resources, and/or inefficiency may affect performance, and/orincrease latency of operations on network 8. To address these issues,data durability block 206 may be configured and/or equipped, in someexamples, to process Galois Field mathematical operations efficiently,and may include specialized circuitry or logic that enables efficientperformance of operations involved in encoding and/or decoding ReedSolomon erasure codes. In examples described herein, one or more ofservers 12 may effectively offload, to access node 17 (or to datadurability block 206), some or all of the computing operations thatmight otherwise be performed by one or more of servers 12 to implementan erasure coding system. By offloading the performance of suchoperations to access node 17, each of servers 12 may operate moreefficiently. In some examples, access node 17 may perform datadurability operations (data encoding, data decoding, and recovery) as atransparent process on network traffic (e.g., transparent to cores 140of FIG. 2 and/or cores 182 of FIG. 4).

FIG. 7A is a conceptual diagram illustrating a matrix operation that maybe used for encoding data pursuant to a Reed-Solomon data reliabilityscheme, in accordance with one or more aspects of the presentdisclosure. FIG. 7A illustrates input vector 702 and matrix 704. Inputvector 702 includes a number of data fragments (e.g., corresponding todata fragments 802D in FIG. 6C) generated by splitting data (e.g., data801) into a number of equal-sized fragments. Matrix 704 has dimensionsof n×(n+m). The upper portion of matrix 704 includes an identity matrix,and the lower portion of matrix 704 includes an encoding matrix.

In the example of FIG. 7A, an in accordance with one or more aspects ofthe present disclosure, access node 17 may encode data. For instance, inan example that can be described with reference to FIG. 6C and FIG. 7A,data durability block 206 performs a Reed Solomon encode operation bymultiplying input vector 702 and matrix 704 to generate output vector706. Data durability block 206 performs such a matrix multiplicationoperation pursuant to known principles of Galois Field mathematics. Theresulting output vector 706 includes output data blocks 708 (e.g.,corresponding to data fragments 802D) and output parity blocks 709(e.g., corresponding to data fragments 802P) of FIG. 6C. Each of outputparity blocks 709 may be the result of a vector dot product of inputvector 702 with coefficients in matrix 704. In some examples, datadurability block 206 may perform a modified and/or streamlined matrixmultiplication operation that takes advantage of certain efficienciesresulting from the fact that a portion of matrix 704 is, as in FIG. 7A,an identity matrix.

FIG. 7B is a conceptual diagram illustrating a matrix operation that maybe used for decoding data that has been encoded pursuant to aReed-Solomon data reliability scheme, in accordance with one or moreaspects of the present disclosure. FIG. 7B illustrates recovery vector722 and matrix 704. Modified recovery vector 722′ includes a subset ofthe data blocks and parity blocks from the original input vector (i.e.,vector 722). In the example shown, recovery vector 722′ includes onlyD0, D2, D4, P0, and P1. Matrix 724 is derived from matrix 704 based onthe subset of data blocks included in modified recovery vector 722′, andhas dimensions corresponding to the size of modified recovery vector722′.

In the example of FIG. 7B, an in accordance with one or more aspects ofthe present disclosure, access node 17 may reconstruct, from modifiedrecovery vector 722′, input vector 702 shown in FIG. 7A (assuming thatin FIG. 7A, n=5 and a 5+3 Reed Solomon scheme is being used). Forinstance, with reference to FIG. 6C and FIG. 7B, data durability block206 identifies the available data blocks corresponding to input vector702. In the example of FIG. 7B, the available data blocks are D0, D2,and D4. Data durability block 206 determines that D1 and D3 are notavailable. Data durability block 206 also identifies available parityblocks, and in the example of FIG. 7B, identifies P0 and P1 as beingavailable. Data durability block 206 constructs modified recovery vector722′ from the available data and parity blocks. Data durability block206 generates matrix 724 by selecting rows from the original encodematrix 704. The rows selected by data durability block 206 are based onthe data and/or parity blocks that are or are not available. Datadurability block 206 processes matrix 724 to generate decode matrix724′, and performs a vector dot product of modified recovery vector 722′and decode matrix 724′ to generate the original input vector, using, forexample, known principles of Galois Field mathematics. As a result, datadurability block 206 generates vector 726, which would be the same asinput vector 702 from FIG. 7A, assuming n=5 in FIG. 7A.

FIG. 7C is a conceptual diagram illustrating a strided read operationthat may be used to generate parity blocks, in accordance with one ormore aspects of the present disclosure. The example of FIG. 7Cillustrates a Reed Solomon 6+4 encoding operation in which six datafragments 802D-1 through 802D-6 (collectively “data fragments 802D”) areused to generate four parity data fragments 802P-1 through 802-4(collectively “data fragments 802P”). To perform the encoding operation,a matrix multiplication operation is performed involving data fragments802D and coefficient matrix 804, which results in data fragments 802P.

FIG. 7C illustrates data fragments 802D arranged within storage 810,which may be storage at one or more locations within network 8, orstorage within cache memory of accelerator 200 of FIG. 5, memoryassociated with a DMA inline accelerator, or other storage. In someexamples, reading from storage 810 may be a costly and/or time-consumingoperation, so it may be beneficial to limit the number times that readoperations involving storage 810 are performed. However, during a ReedSolomon encode operation performed by data durability block 206 (e.g.,as shown in FIG. 6C and FIG. 7A), generating each of data fragments 802Pmay involve reading segments of data from each of data fragments 802Dmultiple times. Accordingly, at least some implementations forgenerating data fragments 802P from data fragments 802D may involve manyread operations.

To reduce the number of read operations performed involving datafragments 802D, data durability block 206 may, in some examples, readthe data from each of data fragments 802D only a single time insuccession, perform the matrix multiplication operations after readingthe data for each of data fragments 802D, and then generate a partialresult for each of data fragments 802D. Data durability block 206 maythereafter combine the partial results to generate each of datafragments 802P. However, while the number of times that read operationsinvolving data fragments 802D may be reduced in such an implementation,the storage required for the partial results generated may besignificant. Further, combining the partial results to generate each ofdata fragments 802P may also involve many read operations. If thepartial solutions are stored in cache memory, a large amount of cachememory is required, which might not be an optimal solution. If thepartial solutions are not stored in cache memory, but instead, arestored in secondary storage (e.g., on disk) then such an implementationis also not optimal, since combining the partial results to generateeach of data fragments 802P may also involve many (relatively slowand/or costly) read operations.

In another example, illustrated in FIG. 7C, reducing the number of readoperations when generating data fragments 802P from data fragments 802Dmay be achieved through a strided read operation involving readingportions of each of data fragments 802D. In some examples, performing a“strided” read operation involves reading a portion of data fragment802D-1, and then jumping (by an offset or “stride” distance) to datafragment 802D-2, and then reading a corresponding portion of datafragment 802D-2. The process is continued until a corresponding portionof each of data fragments 802D has been read. In other words, afterreading a portion of data fragment 802D-1 referenced by a pointer, thepointer is advanced a “stride” amount to data fragment 802D-2, acorresponding read is performed, and then the pointer is again advancedanother stride to data fragment 802D-3. The process continues until thepointer references and reads from the final data fragment 802D-6. Theeffect of the strided read operations is that each of data fragments802P fragments can be generated by using just a portion of each block,and by reading only the required portion of each of data fragments 802D(which happens to correspond to each of the strided read operations),each of data fragments 802P can be created in succession with a reducednumber of read operations, and requiring relatively little or nointermediate storage.

Accordingly, data durability block 206 may generate data fragments 802Pthrough a strided read operation of data fragments 802D. For instance,in the example of FIG. 7C, and with reference to FIG. 6C, datadurability block 206 accesses a pointer that references a subset of datafrom data fragment 802D-1. Data durability block 206 reads thereferenced data and performs a matrix multiplication operation pursuantto an erasure coding encode operation. Data durability block 206 thenupdates the pointer to reference a subset of data from data fragment802D-2. Data durability block 206 reads the referenced data from datafragment 802D-2, effectively jumping from data fragment 802D-1 to datafragment 802D-2, rather than reading data fragment 802D-1 serially.After each read operation, data durability block 206 performs a matrixmultiplication using the read data, and then updates the pointer topoint to a subset of data within the next data fragments 802D. Thisprocess continues until a subset of data from each of data fragments802D has been accessed, and an appropriate matrix multiplicationoperation has been performed. Data durability block 206 may repeatsimilar read operations involving other subsets of each of datafragments 802D until sufficient data is read to generate one of datafragments 802P. Data durability block 206 then generates one of datafragments 802P, and then begins a new set of strided read operations togenerate another one of data fragments 802P in a similar manner. Thisprocess is completed until all of data fragments 802P are generated. Insuch an implementation, it may be possible to read each subset of datafrom each of data fragments 802D only a single time (or a small numberof times), but the amount of intermediate storage required to generateeach of data fragments 802P is also reduced.

Performing some types of erasure coding operations, such as Reed Solomonerasure coding, may involve applications of Galois Field mathematics. InGalois Field arithmetic, add and subtract operations are simply XORoperations, but multiplication and division are complicated, and oftennot well supported by commercial processing circuitry. Division may beperformed by multiplication operations with the inverse value.Multiplication operations may be performed in one of a few differentways. In one option, the numbers or parameters to be multiplied areviewed as polynomials, and are multiplied accordingly. In anotheroption, a table lookup is used, but the size of the table growsexponentially with the size of the values being multiplied. For someword sizes, the size of the table may be impractical and/or prohibitive.

In yet other options, it may be possible to translate multiplicationoperations into XOR operations of elements. In one such option,multiplication operations may be performed using a much smaller table,taking advantage of the observation that a multiplication operation,such as 0xAB*0X34 may be rewritten as 0xAB*0x30+0xAB*0x04. Such a tablemay have a size suitable for practical implementations. In a second suchoption, two lookup tables are used, one for each static coefficient, of16 values each. One of the tables is used to multiply values with thelower portion of the data byte (0x01, 0x02, 0x03 . . . 0x0F). The othertable is used to multiply values with the upper portion of the data byte(0x10, 0x20, 0x30 . . . 0xF0). Using one or more of such techniques forGalois Field multiplications, operations may be performed as a statelessoperation, suitable for a DMA inline accelerator. Such operations may becapable of being generalized as vector dot product operations in GaloisField, using two lookups in two 16-byte tables, followed by XORoperations. In some examples, as each of data fragments 802P isgenerated, data durability block 206 outputs one of data fragments 802Pto scatter block 212 of FIG. 5. Scatter block 212 outputs data fragments802P to access node 17. Access node 17 writes or stores each of datafragments 802P in a different failure domain within network 8 asillustrated in FIG. 6C.

In FIG. 7C, and in some examples, matrix 804 may be stored in on chipmemory, within accelerator 200, for efficient parity calculation. Forsmaller block sizes this might result in latency. To counter this,accelerator 200 may support two modes: (1) LOAD Only, and (2) LOAD andCOMPUTE.

The LOAD only mode provides a method to keep the coefficient in SRAM andsoftware executing on access node 17 (or data durability block 206) mayissue multiple encode commands based on loaded coefficient. The LOAD andCOMPUTE method provides a mechanism to pass data blocks along with thecoefficient matrix and perform the compute. Data durability block 206may, in some examples, provide a fully software addressable memory inwhich software can load multiple coefficient matrices and partitionmemory for LOAD only and LOAD and COMPUTE. This allows a flexibility forsoftware to work in both modes in parallel on different data sets.

FIG. 7D is a conceptual diagram illustrating a matrix operation that maybe used for encoding and/or decoding data pursuant to a parity arraycoding scheme, in accordance with one or more aspects of the presentdisclosure. The example of FIG. 7D illustrates that other datadurability schemes and/or erasure coding algorithms, beyond Reed Solomonschemes, such as parity array codes, can be implemented using matrixoperations. In general, in some parity array erasure coding schemes,each of the parity blocks is generated using data from only a subset ofthe data blocks (rather than, for example, generating the parity blocksbased on data from all of the data blocks). For example, for erasurecodes based on the EvenOdd coding algorithm, the X Code algorithm, andHoVer Codes algorithm, parity blocks are generated based on subsets ofdata blocks. Generating parity blocks for such erasure coding schemestherefore may involve selection of subsets of the data blocks, which isan operation that matrices can be used to perform. By choosing orgenerating an appropriate matrix, as illustrated in FIG. 7D, the datablocks or fragments used to generate each of the parity blocks can beselected.

Accordingly, data durability block 206 may apply a common matrixapproach to both encoding and decoding parity blocks for a wide varietyof erasure coding schemes. For instance, in the example of FIG. 7D, andwith reference to FIG. 6C, data durability block 206 identifies, throughinput or a configuration setting, the type of erasure coding schemebeing used. If data durability block 206 determines that the ReedSolomon algorithm is being used, data is encoded and/or decoded in themanner illustrated in FIG. 7A, FIG. 7B, and FIG. 7C. If data durabilityblock 206 determines that a different type of erasure coding algorithmis being used, such as a parity array coding algorithm data durabilityblock 206 generates or accesses a matrix having the form of matrix 744.Matrix 744 may be specific to the particular algorithm being used, andis chosen (or generated) based on the combinations of data blocks thatthe algorithm uses to generate parity blocks.

For instance, in the example of FIG. 7D, the erasure coding algorithmbeing employed uses D0, D2, and D4 to generate a first parity blockwithin output vector 746. Further, the erasure coding algorithm uses D1and D2 to generate a second parity block within output vector 746, anduses D3 and D4 to generate a third parity block within output vector746. Accordingly, matrix 744 is configured or chosen to ensure thatthese combinations of data blocks result from a vector dot product ofinput vector 742 and matrix 744, effectively operating as a bit map thatselects which of the data blocks to include in exclusive or operations(XOR) performed by data durability block 206. Once matrix 744 is chosenbased on the algorithm in use, data durability block 206 performs thevector dot product of input vector 742 and matrix 744, which results inoutput vector 746.

By selecting an appropriate coefficient matrix, a variety of erasurecoding algorithms can be implemented through matrix operations, therebyenabling some or all erasure coding algorithms to be performed byapplying a common matrix approach. In such an implementation, thecoefficient matrix may differ from algorithm to algorithm, but some orall of the matrix operations may be implemented or performed in the sameway. Further, in some or all cases, the same matrix 744 may be used forboth encoding and decoding operations, and matrix 744 may latertherefore be used to reconstruct input vector 742 from a subset of thedata blocks and the parity blocks generated earlier using matrix 744.

FIG. 8A is a block diagram illustrating example operations on a networkin which stored data fragments are read from locations on the network,in accordance with one or more aspects of the present disclosure. In theexample of FIG. 8A, it is assumed that access node 17 has previouslyapplied a 4+2 erasure coding algorithm to split data 801 into four datafragments 802D, generated two data fragments 802P, and stored each ofdata fragments 802D and data fragments 802P across network 8 in themanner illustrated. Specifically, data fragment 802D-1 has been storedat rack 70-9, data fragment 802D-2 has been stored at rack 70-6, datafragment 802D-3 has been stored at rack 70-12, data fragment 802D-4 hasbeen stored at rack 70-3. Further, data fragment 802P-1 has been storedat rack 70-5 and data fragment 802P-2 has been stored at rack 70-14.

In the example of FIG. 8A, and in accordance with one or more aspects ofthe present disclosure, access node 17 may perform a read operation andreconstruct data 801D from data fragments 802D. For instance, in theexample of FIG. 8A, access node 17 receives a request for data 801.Access node 17 accesses a directory, stored within access node 17 orelsewhere, identifying where each of data fragments 802D are stored.Access node 17 determines whether each of data fragments 802D areavailable. Access node 17 determines that each of data fragments 802Dare available, and accesses and/or retrieves each of data fragments 802Dat the locations illustrated in FIG. 8A. Since each of data fragments802D are available, access node 17 does not need to access any of datafragments 802P. Access node 17 reconstructs data 801 by combining eachof data fragments 802D. In some examples, access node 17 may be able toreassemble data 801 by simply stitching each of data fragments 802Dtogether to form data 801, without having to perform decoding operationspursuant to the data durability scheme implemented in network 8. Accessnode 17 responds to the request for data 801 by outputting thereassembled data 801.

FIG. 8B is a block diagram illustrating an example read operation on anetwork in which some data fragments stored on the network areunavailable, in accordance with one or more aspects of the presentdisclosure. As in FIG. 8A, in FIG. 8B, it is assumed that access node 17has split data 801 into four data fragments 802D, has generated two datafragments 802P, and has stored each of data fragments 802D and datafragments 802P across network 8 in the manner illustrated. However, dueto a hardware or software error, or due to some other reason, datafragment 802P-1 stored at rack 70-5 is not available, and data fragment802D-1 stored at rack 70-9 is not available.

In the example of FIG. 8B, and in accordance with one or more aspects ofthe present disclosure, access node 17 may perform a read operation andreconstruct data 801D from available data fragments, including one ormore of data fragments 802D and data fragments 802P. For instance, inthe example of FIG. 8A, access node 17 receives a request for data 801.Access node 17 accesses a directory, stored within access node 17 orelsewhere, identifying where each of data fragments 802D are stored.Access node 17 determines that data fragment 802D-2, data fragment802D-3, and data fragment 802D-4 are available, but that data fragment802D-1, stored at rack 70-9, is not available. Access node 17 accessesand/or retrieves data fragment 802D-2, data fragment 802D-3, and datafragment 802D-4 over network 8. Access node 17 further accesses and/orretrieves data fragment 802P-2 over network 8. Access node 17 passes theaccessed data to accelerator 200. Accelerator 200 causes data durabilityblock 206 to reconstruct data 801 from the accessed data. In someexamples, data durability block 206 performs erasure decoding operations(e.g., a Reed Solomon 4+2 decode operation) to reconstruct data 801.Access node 17 responds to the request for data 801 by outputting thereconstructed data 801.

FIG. 9A is a flow diagram illustrating inline reliability codingoperations performed by an example access node in accordance with one ormore aspects of the present disclosure. FIG. 9A is described belowwithin the context of access node 17 of FIG. 6C and FIG. 8B. In otherexamples, operations described in FIG. 9A may be performed by one ormore other components, modules, systems, or devices. Further, in otherexamples, operations described in connection with FIG. 9A may be merged,performed in a difference sequence, omitted, or may encompass additionaloperations not specifically illustrated or described.

In the example of FIG. 9A, and in accordance with one or more aspects ofthe present disclosure, access node 17 may identify data to be stored(901). For instance, in an example that can be described in connectionwith FIG. 6C, access node 17 receives, from server 12, data 801. Accessnode 17 identifies data 801 as data intended for storage.

Access node 17 may produce a plurality of data fragments (902). Forinstance, in some examples, logic within access node 17 outputs data 801to accelerator 200. Within accelerator 200, data durability block 206(see FIG. 5) processes data 801 by splitting data 801 into data fragment802D-1, data fragment 802D-2, data fragment 802D-3, and data fragment802D-4 (collectively “data fragments 802D”). In addition, datadurability block 206 generates, using a Reed-Solomon erasure codealgorithm, data fragment 802P-1 and data fragment 802P-2 (collectively“data fragments 802P”), which are parity blocks that may be used torecover one or more of data fragments 802D if one or more of datafragments 802D are unavailable.

Access node 17 may store the data fragments on the network (903). Forinstance, in some examples, access node 17 chooses a set of failuredomains for storing each of data fragments 802D and data fragments 802P.In the example of FIG. 6C, each of racks 70 is defined as a failuredomain. Access node 17 stores each of data fragments 802D and datafragments 802P in a different rack 70, as shown in FIG. 6C. In otherexamples, a failure domain may be defined more narrowly, such that afailure domain is a sub-rack or logical rack within each of racks 70. Insuch an example, more than one of data fragments 802D or data fragments802P may be stored on the same rack 70. In still other examples, afailure domain may be defined more broadly, so that a failure domainencompasses more than one racks 70, so that no more than one datafragments 802D or data fragments 802P are stored within the same set ofracks 70 that compose a failure domain.

Access node 17 may receive a request to access data stored on thenetwork (904). For instance, now referring to FIG. 8B, and some periodof time after the data fragments corresponding to data 801 are stored onthe network, access node 17 receives a request to access data 801. Insome examples, the request may originate from the same server 12 thatoriginally sent the data to access node 17. In other examples, therequest may originate from a different server or from a differentdevice.

Access node 17 may determine whether one or more of data fragments 802Dare available (905). For instance, with reference to FIG. 8B, accessnode 17 accesses a directory identifying where each of data fragments802D are stored. Access node 17 determines whether each of datafragments 802D stored at the identified location is available. If accessnode 17 determines that each of data fragments 802D are available,access node 17 accesses each of data fragments 802D at their respectiveidentified location (906). In some situations, however, access node 17determines that at least one of data fragments 802D is not available. Inthe example of FIG. 8B, access node 17 determines that data fragment802D-1 and data fragment 802P-1 are both unavailable. One or more ofdata fragments 802D may be unavailable for a number of reasons,including hardware or software error, or due to temporarilyunavailability of one or more of racks 70 or portions of one or more ofracks 70.

Access node 17 may identify a plurality of available fragments (907).For instance, still referring to FIG. 8B, access node 17 determines thelocation of one or more data fragments 802P. Access node 17 identifiesdata 801D-2, data 801D-3, data 801D-4, and data fragment 802P-2 asavailable data fragments.

Access node 17 may retrieve the available fragments over the network(908). For instance, in the example of FIG. 8B, access node 17retrieves, over switch fabric 14 or otherwise, each of the fouravailable fragments. Access node 17 may store each of the availablefragments in memory within accelerator 200.

Access node 17 may generate data corresponding to data 801 from theavailable fragments (909). For instance, since in the example of FIG. 8Baccess node 17 can reconstruct data 801 based on any four of the datafragments in a collection consisting of data fragments 802D and datafragments 802P, access node 17 reconstructs data 801 from the availablefragments. To do so, data durability block 206 of access node 17processes the stored fragments (e.g., through an appropriate matrixoperation pursuant to a 4+2 Reed-Solomon encoding scheme), and generatesdata 801. Data durability block 206 outputs the reconstructed data 801to accelerator 200, which makes it available to access node 17.

Access node 17 may respond to the request to access the data (910). Forinstance, in some examples, access node 17 outputs data 801 to thedevice that requested data 801, such as server 12.

FIG. 9B is a flow diagram illustrating flexible data durability codingoperations performed by an example access node in accordance with one ormore aspects of the present disclosure. FIG. 9B is described belowwithin the context of access node 17 of FIG. 6C and FIG. 8B. In otherexamples, operations described in FIG. 9B may be performed by one ormore other components, modules, systems, or devices. Further, in otherexamples, operations described in connection with FIG. 9B may be merged,performed in a difference sequence, omitted, or may encompass additionaloperations not specifically illustrated or described.

In the example of FIG. 9B, and in accordance with one or more aspects ofthe present disclosure, access node 17 may identify a reliability scheme(931). For instance, in an example that can be described in connectionwith FIG. 6C and FIG. 8B, access node 17 receives informationidentifying a reliability scheme that is used in network 8 and that hasbeen chosen for use when storing data 801. In some examples, access node17 receives the information in the form of input from an administrator,or access node 17 may access a configuration file, or access node 17 maydetect an appropriate reliability scheme by observing operations withinnetwork 8, or access node 17 may identify a reliability scheme inanother manner.

Access node 17 may determine a coefficient matrix (932). For instance,in some examples, access node 17 may determine, based on the identifiedreliability scheme, an appropriate coefficient matrix that can be usedto implement the identified reliability scheme. In some examples, thecoefficient matrix may be used to both encode and decode parity blocksor fragments derived from data 801. If the reliability scheme is asimple parity scheme, for example, access node 17 may generate or accessa coefficient matrix consistent with the operations described inconnection with FIG. 6A. If the reliability scheme is a Parity ArrayCodes erasure coding scheme, access node 17 may generate and use, forencoding and/or decoding operations, a coefficient matrix consistentwith that described in connection with FIG. 7D. If the reliabilityscheme is a Reed-Solomon reliability scheme, access node 17 may generateor access a coefficient matrix consistent with the matrix described inconnection with FIG. 7A, FIG. 7B, and FIG. 7C. In the example of FIG.6C, a Reed-Solomon 4+2 reliability scheme is used, so access node 17generates a coefficient matrix appropriate for that reliability scheme.

Access node 17 may store the coefficient matrix (933). For instance, insome examples, access node 17 stores the coefficient matrix used for theidentified reliability scheme in memory so that it can be accessed whenencoding and/or decoding data 801. Access node 17 may store thecoefficient matrix in memory unit 183 (see FIG. 4A), or in otherexamples, access node 17 may store the coefficient matrix within storageincluded within accelerator 200 and/or data durability block 206 (seeFIG. 5).

Access node 17 may receive data intended for storage (934). Forinstance, in some examples, access node 17 receives, from server 12,data 801. Access node 17 identifies data 801 as data intended forstorage.

Access node 17 may produce data fragments 802P (935). For instance, inthe example of FIG. 6C, access node 17 outputs data 801 to accelerator200. Within accelerator 200, data durability block 206 (see FIG. 5)processes data 801 in a manner consistent with the identifiedreliability scheme. In the example of FIG. 6C, a Reed-Solomon 4+2reliability scheme is used, so accelerator 200 splits data 801 into datafragment 802D-1, data fragment 802D-2, data fragment 802D-3, and datafragment 802D-4 (collectively “data fragments 802D”). In addition, datadurability block 206 generates, using the stored coefficient matrix,data fragment 802P-1 and data fragment 802P-2 (collectively “datafragments 802P”). Access node 17 may store data fragments 802D and datafragments 802P in the manner illustrated in FIG. 6C.

Access node 17 may receive a request for data (936). For instance,referring now to FIG. 8B, access node 17 waits for a request for data,and some period of time after the parity data for data 801 is produced,access node 17 receives a request to access data 801. The request mayoriginate from server 12, a different server, or from a differentdevice.

Access node 17 may generate data corresponding to data 801 using theparity data (937). For instance, again referring to FIG. 8B, access node17 reconstructs data 801 based on any four of the data fragments in acollection consisting of data fragments 802D and data fragments 802P,all of which are stored on different racks 70 in FIG. 8B. In the exampleof FIG. 8B, data durability block 206 of access node 17 reconstructsdata 801 from data fragments 802D and data fragments 802P by readingdata from network 8. If any of data fragments 802D are unavailable, datadurability block 206 uses one or more of data fragments 802P toreconstruct data 801. To reconstruct the data, data durability block 206uses available data fragments 802D and/or data fragments 802P andgenerates, by applying the stored coefficient matrix, data 801. Datadurability block 206 outputs the reconstructed data 801 to accelerator200, which makes it available to access node 17. Access node 17 outputsdata 801 to the device that requested data 801.

For processes, apparatuses, and other examples or illustrationsdescribed herein, including in any flowcharts or flow diagrams, certainoperations, acts, steps, or events included in any of the techniquesdescribed herein can be performed in a different sequence, may be added,merged, or left out altogether (e.g., not all described acts or eventsare necessary for the practice of the techniques). Moreover, in certainexamples, operations, acts, steps, or events may be performedconcurrently, e.g., through multi-threaded processing, interruptprocessing, or multiple processors, rather than sequentially. Furthercertain operations, acts, steps, or events may be performedautomatically even if not specifically identified as being performedautomatically. Also, certain operations, acts, steps, or eventsdescribed as being performed automatically may be alternatively notperformed automatically, but rather, such operations, acts, steps, orevents may be, in some examples, performed in response to input oranother event.

The detailed description set forth above is intended as a description ofvarious configurations and is not intended to represent the onlyconfigurations in which the concepts described herein may be practiced.The detailed description includes specific details for the purpose ofproviding a sufficient understanding of the various concepts. However,these concepts may be practiced without these specific details. In someinstances, well-known structures and components are shown in blockdiagram form in the referenced figures in order to avoid obscuring suchconcepts.

In accordance with one or more aspects of this disclosure, the term “or”may be interrupted as “and/or” where context does not dictate otherwise.Additionally, while phrases such as “one or more” or “at least one” orthe like may have been used in some instances but not others; thoseinstances where such language was not used may be interpreted to havesuch a meaning implied where context does not dictate otherwise.

In one or more examples, the functions described may be implemented inhardware, software, firmware, or any combination thereof. If implementedin software, the functions may be stored, as one or more instructions orcode, on and/or transmitted over a computer-readable medium and executedby a hardware-based processing unit. Computer-readable media may includecomputer-readable storage media, which corresponds to a tangible mediumsuch as data storage media, or communication media including any mediumthat facilitates transfer of a computer program from one place toanother (e.g., pursuant to a communication protocol). In this manner,computer-readable media generally may correspond to (1) tangiblecomputer-readable storage media, which is non-transitory or (2) acommunication medium such as a signal or carrier wave. Data storagemedia may be any available media that can be accessed by one or morecomputers or one or more processors to retrieve instructions, codeand/or data structures for implementation of the techniques described inthis disclosure. A computer program product may include acomputer-readable medium.

By way of example, and not limitation, such computer-readable storagemedia can include RAM, ROM, EEPROM, CD-ROM or other optical diskstorage, magnetic disk storage, or other magnetic storage devices, flashmemory, or any other medium that can be used to store desired programcode in the form of instructions or data structures and that can beaccessed by a computer. Also, any connection is properly termed acomputer-readable medium. For example, if instructions are transmittedfrom a website, server, or other remote source using a coaxial cable,fiber optic cable, twisted pair, digital subscriber line (DSL), orwireless technologies such as infrared, radio, and microwave, then thecoaxial cable, fiber optic cable, twisted pair, DSL, or wirelesstechnologies such as infrared, radio, and microwave are included in thedefinition of medium. It should be understood, however, thatcomputer-readable storage media and data storage media do not includeconnections, carrier waves, signals, or other transient media, but areinstead directed to non-transient, tangible storage media. Disk anddisc, as used, includes compact disc (CD), laser disc, optical disc,digital versatile disc (DVD), floppy disk and Blu-ray disc, where disksusually reproduce data magnetically, while discs reproduce dataoptically with lasers. Combinations of the above should also be includedwithin the scope of computer-readable media.

Instructions may be executed by one or more processors, such as one ormore digital signal processors (DSPs), general purpose microprocessors,application specific integrated circuits (ASICs), field programmablelogic arrays (FPGAs), or other equivalent integrated or discrete logiccircuitry. Accordingly, the terms “processor” or “processing circuitry”as used herein may each refer to any of the foregoing structure or anyother structure suitable for implementation of the techniques described.In addition, in some examples, the functionality described may beprovided within dedicated hardware and/or software modules. Also, thetechniques could be fully implemented in one or more circuits or logicelements.

The techniques of this disclosure may be implemented in a wide varietyof devices or apparatuses, including a wireless handset, a mobile ornon-mobile computing device, a wearable or non-wearable computingdevice, an integrated circuit (IC) or a set of ICs (e.g., a chip set).Various components, modules, or units are described in this disclosureto emphasize functional aspects of devices configured to perform thedisclosed techniques, but do not necessarily require realization bydifferent hardware units. Rather, as described above, various units maybe combined in a hardware unit or provided by a collection ofinteroperating hardware units, including one or more processors asdescribed above, in conjunction with suitable software and/or firmware.

What is claimed is:
 1. A method comprising: generating from a set ofdata, by data durability circuitry within a data processing unit, aplurality of data fragments, wherein generating the plurality of datafragments includes generating a plurality of data split fragments bysplitting the set of data, and generating a plurality of parityfragments by applying an erasure coding algorithm, and whereingenerating the plurality of parity fragments includes performing aplurality of strided read operations across the plurality of data splitfragments, wherein the plurality of strided read operations includesreading, for each of the plurality of data split fragments, data at astride offset relative to the beginning of each of the data splitfragments; storing, by the data processing unit, the plurality of datafragments, wherein at least some of the plurality of data fragments isstored within a different fault domain, wherein each fault domainincludes one or more hardware subsystems not included within any otherof the fault domains; receiving, by the data processing unit and from arequesting device, a request to access at least a portion of the set ofdata; determining, by the data processing unit, that one or more of theplurality of data fragments is not available; identifying, by the dataprocessing unit, a plurality of available data fragments, wherein theplurality of available data fragments is a subset of the plurality ofdata fragments; retrieving, by the data processing unit, at least asubset of the plurality of available data fragments over a network;generating, by the data durability circuitry within the data processingunit, a reconstructed set of data from the available data fragments; andresponding to the request by outputting, by the data processing unit,the reconstructed data.
 2. The method of claim 1, wherein the set ofdata is from one or more of a plurality of servers, and wherein therequesting device is one of the plurality of servers.
 3. The method ofclaim 1, wherein storing the plurality of data fragments includes:storing each of the plurality of data fragments within a different faultdomain on the network.
 4. The method of claim 1, wherein generating theplurality of data fragments includes: generating a plurality of datasplit fragments by splitting the set of data pursuant to a RAID encodingscheme, and generating a plurality of parity fragments by applying anerasure coding algorithm.
 5. The method of claim 1, wherein generatingthe plurality of parity fragments further includes: storing, withincontiguous memory and before performing the plurality of strided readoperations, each of the plurality of data split fragments.
 6. The methodof claim 1, wherein generating the plurality of parity fragments furtherincludes: generating, as a result of each sequence of the strided readoperations, one of the plurality of parity fragments without modifyingany of the other plurality of parity fragments.
 7. The method of claim1, wherein the erasure coding algorithm is a Reed Solomon erasure codingalgorithm.
 8. The method of claim 1, wherein identifying the pluralityof available data fragments includes identifying at least as manyavailable data fragments as there are data split fragments.
 9. Themethod of claim 1, wherein storing the plurality of data fragmentsincludes: storing all of the plurality of data fragments within adifferent fault domain on the network.
 10. The method of claim 1,wherein storing the plurality of data fragments includes: storing eachof the plurality of data fragments within a different rack.
 11. Themethod of claim 1, wherein a fault domain is defined as at least one ofthe following: a network storage compute unit, a logical rack, or aphysical rack.
 12. The method of claim 1, wherein at least one of thestored plurality of data fragments is unavailable as a result of anadministrative operation.
 13. The method of claim 12, wherein theadministrative operation is a software upgrade operation.
 14. The methodof claim 1, wherein the data durability circuitry within the dataprocessing unit includes circuitry for efficiently performing matrixmultiplication operations, and wherein generating the reconstructed setof data includes: performing, by the circuitry, matrix multiplicationoperations.
 15. The method of claim 1, wherein the data durabilitycircuitry within the data processing unit includes circuitry forefficiently performing Galois Field mathematics operations, and whereingenerating the reconstructed set of data includes: performing, by thecircuitry, Galois Field mathematics operations.
 16. The method of claim15, wherein performing the Galois Field mathematics operations includeperforming XOR operations and using at least one lookup table.
 17. Adata processing unit comprising: at least one memory device; datadurability circuitry; and processing circuitry capable of controllingthe data durability circuitry and having access to the at least onememory device and configured to: cause the data durability circuitry togenerate, from a set of data, a plurality of data fragments, whereingenerating the plurality of data fragments includes generating aplurality of data split fragments by splitting the set of data, andgenerating a plurality of parity fragments by applying an erasure codingalgorithm, and wherein to generate the plurality of parity fragments,the data durability circuitry performs a plurality of strided readoperations across the plurality of data split fragments, wherein theplurality of strided read operations includes reading, for each of theplurality of data split fragments, data at a stride offset relative tothe beginning of each of the data split fragments, store the pluralityof data fragments on a network, wherein at least some of the pluralityof data fragments is stored within a different fault domain on thenetwork, receive, from a requesting server, a request to access at leasta portion of the set of data, wherein the requesting server is one ofthe plurality of servers, determine that one or more of the plurality ofdata fragments is not available on the network, identify a plurality ofavailable data fragments, wherein the plurality of available datafragments is a subset of the plurality of data fragments, retrieve eachof the plurality of available data fragments over the network, cause thedata durability circuitry to generate a reconstructed set of data fromthe available data fragments, and respond to the request by outputtingthe reconstructed data.
 18. A computer-readable storage mediumcomprising instructions that, when executed, configure processingcircuitry to: cause data durability circuitry to generate, from a set ofdata, a plurality of data fragments, wherein generating the plurality ofdata fragments includes generating a plurality of data split fragmentsby splitting the set of data, and generating a plurality of parityfragments by applying an erasure coding algorithm, and whereingenerating the plurality of parity fragments includes performing aplurality of strided read operations across the plurality of data splitfragments, wherein the plurality of strided read operations includesreading, for each of the plurality of data split fragments, data at astride offset relative to the beginning of each of the data splitfragments; store the plurality of data fragments on a network, whereinat least some of the plurality of data fragments is stored within adifferent fault domain on the network; receive, from a requestingserver, a request to access at least a portion of the set of data,wherein the requesting server is one of the plurality of servers;determine that one or more of the plurality of data fragments is notavailable on the network; identify a plurality of available datafragments, wherein the plurality of available data fragments is a subsetof the plurality of data fragments; retrieve each of the plurality ofavailable data fragments over the network, cause the data durabilitycircuitry to generate a reconstructed set of data from the availabledata fragments; and respond to the request by outputting thereconstructed data.